IStore: Towards High Efficiency, Performance, and Reliability in Distributed Data Storage with Information Dispersal Algorithms

Reliability is one of the major challenges for high performance computing and cloud computing. Data replication is a commonly used mechanism to achieve high reliability. Unfortunately, it has a low storage efficiency among other shortcomings. As an alternative to data replication, information dispersal algorithms offer higher storage efficiency, but at the cost of being too computing-intensive for today’s modern processors. This paper explores the possibility of utilizing erasure coding (a form of information dispersal algorithms) for data redundancy while accelerating its operation with GPUs. We evaluate the performance improvements of the erasure coding from the CPU to the GPU, showing a 10X higher throughput for the GPU. With this promising result, we design and implement a distributed data store with erasure coding, called IStore, to demonstrate that erasure coding could serve as a better alternative to traditional data replication in distributed file systems. A performance evaluation is performed on a 32-node cluster to evaluate the scalability and parameter sensitivity of IStore.